Display-pixel and photosensor-element device and method therefor

ABSTRACT

A display-pixel and photosensor-element device for use as a display and a camera, the device comprising a plurality of light-emitting diode (LED) display elements and a plurality of light-sensitive photosensor devices, together fabricated onto an essentially planar surface so as to create a device that can be used as a display and a camera. In an implementation, a plurality of micro-optic structures can be associated with the plurality of light-sensitive devices. In various exemplary implementations, the LEDs may comprise organic light emitting diodes (OLEDs) or stacked organic light emitting diode (SOLED). In an implementation, the light-emitting diode and the light-sensitive device are integrated into a single element. In an implementation, the device is configured to serve as either or both of a color light sensor array and a display comprising color LEDs.

CROSS-REFERENCE TO RELATED PATENT APPLICATIONS

The present application claims the benefit of and is a continuationapplication of U.S. patent application Ser. No. 12/419,229 filed on Apr.6, 2009, which is a Divisional of U.S. patent application Ser. No.09/601,384 filed on Jul. 27, 2000, which was an application filed under35 U.S.C. §371 of International Patent Application No. PCT/US1999/001789filed Jan. 27, 1999, which claims the benefit of U.S. ProvisionalApplication No. 60/072,762 filed on Jan. 27, 1998, the disclosures ofall of which are incorporated by reference herein.

FIELD OF THE INVENTION

The present invention relates generally to multimedia conferencingsystems, and more particularly to multimedia-enabled communication andcomputing devices. Still more particularly, the present invention is adevice for providing real-time multimedia conferencing capabilities toone or more companion computers or on a stand-alone basis.

BACKGROUND

Early computers were large, clumsy, difficult-to-operate and unreliableroom-sized systems shared within a single location. Similarly, earlyvideo and graphics teleconferencing systems suffered from the samedrawbacks, and were also shared within a single location. With regard tocomputers, technological innovations enabled the advent of desktop“personal computers.” Relative to teleconferencing systems, newtechnologies were also introduced, such as those described in U.S. Pat.No. 5,617,539, entitled “Multimedia Collaboration System with SeparateData Network and A/V Network Controlled by Information Transmitting onthe Data Network,” that brought high-quality, reliable video andgraphics teleconferencing capabilities to a user's desktop. In bothearly desktop personal computers and conferencing systems, there wereand remain many incompatible implementations.

Digital technology innovations targeted at working in conjunction withmarket forces gave rise to standardized desktop computer platforms, suchas Microsoft/Intel machines and Apple machines, which have existing andstrengthening unifying ties between them. The standardization ofconverging platforms unified fragmentations that existed within thecomputer hardware and software industries, such that immense economiesof scale lowered the per-desktop development and manufacturing costs.This in turn greatly accelerated desktop computer usage and promoted theinterworking between applications such as work processing, spreadsheet,and presentation tool applications that freely exchange data today. As aresult, businesses employing such interworking applications became moreefficient and productive. The push for greater efficiency has fueled thedevelopment of additional innovations, which further led to developmentssuch as the explosion in electronic commerce as facilitated by theworld-wide Internet.

Relative to present-day desktop conferencing, there are many networkingapproaches characterized by varying audio/video (A/V) quality andscalability. In recent years, customers have assumed a wide range ofpositions in their investments in such technologies. At one end of thisrange, various types of dedicated analog A/V overlay networks exist thatdeliver high-quality A/V signals at a low cost. At another end of thisrange are local area data network technologies such as switched Ethernetand ATM data hubs that function with high-performance desktop computers.These desktop computers and data networking technologies currentlysupport only lower-quality A/V capabilities at a relatively high cost.Despite this drawback, these desktop computers and data networkingtechnologies are believed to be the preferred path for eventuallyproviding high-quality A/V capabilities at a low cost. Other A/Vnetworking solutions, such as ISDN to the desktop, also lie in thisrange.

Within each of many separate networked A/V technology “islands,” variousapproaches toward providing multimedia applications such asteleconferencing, video mail, video broadcast, video conferencerecording, video-on-demand, video attachments to documents and/or webpages, and other applications can be performed only in fragmented wayswith limited interworking capability. For many years, it has beenprojected that the desktop computer industry and the data networkingindustry will solve such fragmentation and interworking problems, andeventually create a unified, low-cost solution. Several generations ofthese technologies and products have consistently fallen short ofsatisfying this long-felt need. Furthermore, it is likely to bedisadvantageous to continue to rely upon the aforementioned industriesto satisfy such needs. For example, if the introduction of today'sstandardized multi-method fax technology had been held back by those whomaintain that the idea that all electronic text should only be computerASCII (as advocated, for example, by M.I.T. Media Lab DirectorNegroponte), a great amount of the fax-leveraged domestic andinternational commerce that has occurred since the early 1980's may nothave occurred. Desktop multimedia technologies and products arecurrently in an analogous position, as it is commonly accepted that itwill be only the desktop computer and data networking industries that atsome point in the future will make high-quality networked A/V widely anduniformly available, and at the same time it is doubtful that this willoccur any time soon.

What is sorely needed, given the pace and market strategies of thedesktop computer and data networking industries, is an integration ofseparate technology and application islands into a single low-cost,manufacturable, reliable real-time multimedia collaboration apparatuscapable of supporting a wide range of A/V networking technologies; A/Vapplications; and A/V and data networking configurations in a widevariety of practical environments. A need also exists for a design orarchitecture that makes such an apparatus readily adaptable to futuretechnological evolution, such that the apparatus may accommodateevolving or new families of interrelated standards.

SUMMARY OF THE INVENTION

The inventive methodology is directed to methods and systems thatsubstantially obviate one or more problems associated with conventionalphoto sensors and display devices.

In accordance with one aspect of the inventive methodology, there isprovided a display-pixel and photosensor-element device for use as adisplay and a camera. The inventive display-pixel andphotosensor-element device incorporates multiple display elements, eachdisplay element including a light-emitting diode; and multiplephotosensor elements, each photosensor element including alight-sensitive device. In the inventive device, the multiple displayelements and multiple photosensor elements are fabricated onto anessentially planar surface so as to create a device that is usable as adisplay and a camera.

Additional aspects related to the invention will be set forth in part inthe description which follows, and in part will be obvious from thedescription, or may be learned by practice of the invention. Aspects ofthe invention may be realized and attained by means of the elements andcombinations of various elements and aspects particularly pointed out inthe following detailed description and the appended claims.

It is to be understood that both the foregoing and the followingdescriptions are exemplary and explanatory only and are not intended tolimit the claimed invention or application thereof in any mannerwhatsoever.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a high-level block diagram of a multimedia collaborationdevice constructed in accordance with the present invention.

FIG. 2 is a high-level perspective view illustrating a box package forthe multimedia collaboration device.

FIG. 3 is a high-level drawing of a plug-in card package for themultimedia collaboration device, which also includes a bus interface.

FIG. 4 is a perspective view of a stand-alone package for the multimediacollaboration device, which includes a camera, a display, a microphonearray, and speakers.

FIG. 5 is a block diagram of a first embodiment of a multimediacollaboration device constructed in accordance with the presentinvention, and which provides primary and auxiliary (AUX) support foranalog audio/video (A/V) input/output (I/O), and further providessupport for networked digital streaming.

FIG. 6 is a block diagram of a second embodiment of a multimediacollaboration device, which provides primary support for analog audioI/O and digital visual I/O, and further supports analog and digitalauxiliary A/V I/O, plus networked digital streaming.

FIG. 7 is a block diagram of a third embodiment of a multimediacollaboration device, which provides primary support for analog audioI/O and digital visual I/O, support for digital auxiliary A/V I/O, andsupport for networked digital streaming.

FIG. 8 is a block diagram of an adaptive echo-canceled stereo microphoneand stereo speaker arrangement within an audio signal conditioning unitof the present invention.

FIG. 9 is a block diagram of an adaptive echo-canceled mono-outputsynthetic aperture microphone arrangement, assuming stereo speakers,within the audio signal conditioning unit, which is of particular valuein noisy environments such as office cubicles or service depot areas.

FIG. 10 is an illustration showing an exemplary localized primaryhot-spot, within which the synthetic aperture microphone has enhancedsensitivity to sound waves produced by a user.

FIG. 11 is an illustration showing exemplary primary hot-spotdirectivity, where the synthetic aperture microphone captures or rejectsdirectionally-specific sound energy from a user within a primaryhot-spot that is offset relative to that shown in FIG. 10.

FIG. 12 is an illustration showing exemplary reflected speech energyrejection by the synthetic aperture microphone.

FIG. 13 is an illustration showing exemplary ambient audio noiserejection by the synthetic aperture microphone.

FIG. 14 is a block diagram of a first embodiment of a first and a secondmultimedia network interface provided by the present invention.

FIG. 15 is a block diagram of a second embodiment of a first and asecond multimedia network interface provided by the present invention.

FIG. 16 is an illustration of a first photosensor and display elementplanar interleaving technique.

FIG. 17 is an illustration of an exemplary photosensor element color anddisplay element color distribution scheme.

FIG. 18 is an illustration of a second alternating photosensor anddisplay element interleaving technique, in which photosensor and displayelement geometries and size differentials aid in minimizing pixel pitchand maximizing displayed image resolution.

FIG. 19 is a cross-sectional view showing a full-color pixel arrayintegrated with a photosensor element array upon a common substrate.

FIG. 20 is a cross-sectional view showing an integrated full-colorpixel/photosensor element, which may form the basis of an integrateddisplay element/photosensor element array.

FIG. 21 is a cross-sectional view of a first full-coloremitter/detector.

FIG. 22 is a cross-sectional view of a second full-coloremitter/detector.

FIG. 23 is a cross-sectional view of a third full-coloremitter/detector.

FIG. 24 is a top-view of an exemplary microoptic layer having differentoptical regions defined therein.

FIG. 25 is an illustration showing individually-apertured photosensorelements capturing light from portions of an object and outputtingsignals to an imaging unit.

DETAILED DESCRIPTION 4.1 General Provisions

The present invention comprises a device that provides analogaudio/video and/or digital audio/visual (both referred to herein as A/V)multimedia collaboration capabilities to a user coupled to a multimedianetwork, such as a Multimedia Local Area Network (MLA/V) as described inU.S. Pat. No. 5,617,539 the disclosure of which is incorporated hereinby reference.

The present invention may operate either in conjunction with one or moreuser's computers or in a stand-alone manner, and may support two-wayvideoconferencing, two-way message publishing, one-way broadcasttransmission or reception, one-way media-on-demand applications, as wellas other audio, video, and/or multimedia functionality or operations.The present invention may support such multimedia functionality across awide range of multimedia network implementations, including mixed analogand digital and/or all-digital multimedia networks. When used inconjunction with a companion computer (i.e., desktop, laptop,special-purpose workstation or other type of computer), the presentinvention may operate as a high-performance multimedia processing devicethat offloads potentially computation-intensive multimedia processingtasks from the companion computer.

The present invention unifies several previously segregated or disparateaudio-, video-, and/or multimedia-related technologies in a singlephysical device that supports multiple multimedia applications andmultiple network signal formats and standards. Such technologies mayinclude hardware and/or software that provide audio signal processing,analog-to-digital (A-D) and digital-to-analog (D-A) conversion,compression and decompression, signal routing, signal level control,video conferencing, stored video-on-demand, internet browsing, messagepublishing, and data networking capabilities. Heretofore, thesetechnologies were typically implemented via separate devices and/orsystems that may have operated in accordance with different data orsignal formats and/or standards, and that offered limited ability (ifany) to interface or operate together.

In particular, the multimedia collaboration device described hereinsupports functionality that may include the following:

1. Audio signal handling:

a) stereo speakers—to provide realistic audio reproduction capabilitiesneeded for multimedia presentations, music, and multiportteleconferencing, including support for three-dimensional sound andaudio positioning metaphors;

b) adaptive echo-canceled stereo speakers for the environment and monoor stereo microphone—to provide high-quality, realistic audiointeractions and eliminate echo and/or feedback in conferencingsituations; and

c) adaptive echo-canceled mono synthetic aperture microphone—tosignificantly improve audio capture performance in noise-prone orpoorly-controlled audio environments, such as office cubicles or publickiosks.

2. One or more data networking protocols, where such protocols may spana range of technological generations. In one embodiment, the presentinvention includes built-in support for 10 and 100 Megabit-per-second(MBPS) Ethernet, as well as Gigabit Ethernet, via Unshielded TwistedPair (UTP) wiring. Other embodiments could include support for other oradditional networking protocols, such as Asynchronous Transfer Mode(ATM) networking and Integrated Services Digital Network (ISDN).

3. One or more analog A/V signal transmission/reception formats, wheresuch formats may span various means of:

a) Analog A/V signal transfer via a separate pair of wires for each ofaudio transmit, audio receive, video transmit, and video receive (i.e.,a total of four sets of UTP wires);

b) Analog A/V signal transfer via a single set of UTP wires foraudio/video transmit, plus a single set of UTP wires for audio/videoreceive (i.e., a total of two twisted-pairs carrying analog A/Vsignals), through frequency modulation (FM) or other multiplexingtechniques;

c) Analog A/V signal transfer via encoding both audio and video signalson a single set of UTP wires, for example, through FM or othermultiplexing methods and perhaps 2-wire/4-wire electronic hybrids; and

d) Any of the above approaches that carry the analog A/V signals on thesame wire pairs as used by data networking circuits (through the use ofFM or other modulation techniques).

Either of the above analog A/V signal transfer formats allow the use ofa single conventional data network connector for carrying both analogA/V and data networking signals. For example, a standard 8-wire RJ-45connector can support 10 and/or 100 MBPS Ethernet in conjunction withanalog A/V signal transfer, using two twisted pairs for Ethernetnetworking and two twisted pairs for A/V signal transfer. In the eventthat data networking is implemented via a protocol for which asufficient number of connector pins or wires are unavailable for A/Vsignal transfer, such as Gigabit Ethernet, which conventionally utilizesthe entire physical capacity of an RJ-45 connector, the presentinvention may include an additional connector or coupling for analog A/Vsignal transfer.

4. Digital multimedia streaming I/O, transmitted to and/or received froma multimedia network and/or a companion computer, as further describedbelow.

5. Internal A/V signal encoding and decoding capabilities to support A/Vcompression formats such as MPEG 1/2/4, JPEG, H.310, H.320, H.323,QuickTime, etc.

6. Internal data routing capabilities, through which data packets,cells, or streams may be selectively transferred among a multimedianetwork, the present invention, and/or a set of companion computers.

7. Multimedia call and connection control protocols, such as describedin U.S. Pat. No. 5,617,539.

8. Internet browsing and multimedia internet message transfercapabilities.

9. Data sharing and/or application sharing protocols.

10. Network configuration and/or network traffic monitoringcapabilities.

Through the combination of the data routing, internal encoding/decoding,and/or digital streaming capabilities, the present invention may operateas a multimedia processing device that offloads potentiallycomputationally-intensive multimedia processing tasks from a companioncomputer. Use of the present invention to reduce a companion computer'sprocessing burden can be particularly advantageous in real-timemultimedia situations. The present invention may further provide anolder or outdated computer with comprehensive real-time multimediacollaboration capabilities, as described below. Additionally, thepresent invention may operate as a stand-alone device, such as aself-contained internet or intranet appliance having real-timemultimedia capabilities, and/or an ISDN video teleconferencing terminal.

The present invention also may advantageously incorporate newtechnologies, including an integrated camera/display device as describedin detail below. Furthermore, the present invention provides support fortechnology and standards evolution by 1) facilitating the use ofstandard plug-in and/or replaceable components, which may be upgraded orreplaced over time; 2) providing designed-in support forrecently-developed technologies that are likely to gain widespread use,such as switched 10 MBPS full-duplex internet, 100 MBPS switchedEthernet, ATM, or Gigabit Ethernet (as well as interim-value networkssuch as ISDN); and 3) providing for upgradability via software and/orfirmware downloads. The present invention may additionally implementparticular capabilities via reconfigurable or reprogrammable logicdevices, such as Field Programmable Gate Arrays (FPGAs). Updatedconfiguration bitstreams can be downloaded into these reconfigurabledevices to provide hardware having upgraded or new capabilities.

4.2 High-Level Architecture and Packaging Options

FIG. 1 is a high-level block diagram of a multimedia collaborationdevice 100 constructed in accordance with the present invention. Themultimedia collaboration device 100 comprises a preamplifier and bufferunit 102; an audio signal conditioning unit 104; a switching unit 106;an Unshielded Twisted Pair (UTP) transceiver 108; a pair splitter 110; arouting unit 112; an encoding/decoding unit 116; a processor set 118; amemory 120; an input device interface 130; a companion computer port136; and a building or premises network port 138.

The premises network port 138 facilitates coupling to premises- orbuilding-based UTP wiring that forms a portion of a multimedia network60. In one embodiment, the premises network port 138 comprises aconventional network coupling, such as an RJ-45 connector. The companioncomputer port 136 facilitates coupling to one or more host or companioncomputers 50, such that the present invention can offload real-timemultimedia processing tasks from a companion computer 50 and/or providea pass-through for data packet exchange between a host computer 50 andthe multimedia network 60. In one embodiment, the companion computerport 136 comprises a conventional network coupling that is compatiblewith the premises network port 138. In another embodiment, the premisesnetwork port 138 may employ a more sophisticated or modern protocol thanthat used by the companion computer port 136. In yet another embodiment,a host or companion computer may access the multimedia collaborationdevice 100 via the premises network port 138, and hence such anembodiment may not include a separate companion computer port 136. It isalso possible for the present invention to communicate with a host orcompanion computer 50 over the data networking ports 136, 138 for use inrunning Graphical User Interfaces (GUIs) or coordinating withapplication processes executing on the host or companion computer 50.

The preamplifier and buffer unit 102 receives A/V signals from a leftand a right microphone 140.1, 140.2 and a camera 142, and transmits A/Vsignals to a left and a right speaker 144.1, 144.2 and a display device146. The preamplifier and buffer unit 102 can additionally send andreceive A/V signals via a set of auxiliary (AUX) A/V ports 148, whichcould couple to a device such as a Video Cassette Recorder (VCR).

As elaborated upon below, the audio signal conditioning unit 104provides volume control functionality in conjunction with echo-canceledstereo microphone or mono synthetic aperture microphone capabilities. Inone embodiment, the echo-canceled stereo microphone and mono syntheticaperture microphone capabilities may be implemented in a singlemode-controlled Digital Signal Processor (DSP) chip, in a manner thatmay facilitate user-selectivity between these two types of microphonefunctionality. If the microphone array 140.1, 140.2 includes more thantwo microphones, it may be desirable to employ DSP techniques tosynthesize a stereo synthetic aperture microphone. Further multiplemicrophone processing modes, such as stochastic noise suppression forextreme noise environments, can also be included.

In the present invention, transfer of incoming and/or outgoing A/Vsignals between a variety of sources and/or destinations is required,including the microphones 140.1, 140.2, the camera 142, the speakers144.1, 144.2, the display device 146, other A/V or I/O devices, thepremises network port 138, and/or the companion computer port 136.Signal transfer pathways for such sources and destinations mayultimately be analog or digital in nature. To meet these switchingneeds, the multimedia collaboration device employs the switching unit106, which selectively routes analog A/V signals associated with themicrophones 140.1, 140.2, the camera 142, the speakers 144.1, 144.2, thedisplay device 146, and/or other devices to or from the analog A/V UTPtransceiver 108 and/or the encoding/decoding unit 116. Theencoding/decoding unit 116 may also perform any required conversionbetween analog and digital formats.

As further described below, the analog A/V UTP transceiver 108 providesan analog signal interface to the pair splitter 110, which separatesdata networking and analog A/V signals. In many cases this signalseparation is most easily accomplished by selectively separating wiresor wire pairs, but may also include the use of passive (or equivalent)wire switching arrangements and programmable Frequency DivisionMultiplexing (FDM) modulators and demodulators. As indicated earlier,the encoding/decoding unit 116 performs conversions between analog anddigital signal formats, and as such also compresses and decompresses A/Vsignals. Although not shown, those skilled in the art will understandthat an ISDN transceiver, inverse multiplexer, network connector, Q.931call control, etc. . . . can be introduced into the architecture to addsupport for ISDN. The processor set 118 controls the operation of themultimedia collaboration device 100, and performs data networkcommunication operations. In conjunction with operating system and othersoftware resident within the memory 120, the processor set 118 mayprovide graphic overlay capabilities on a video image so as to implementany GUI capabilities. These GUIs may facilitate control over theoperations of the present invention, and may further provide internetbrowsing capabilities, as described in detail below. The routing unit112 performs network packet exchange operations between the premisesnetwork port 138, the companion computer port 136, and the processingunit 118, where such packets may include data, portions of, or entiredigital AV streams, and/or network configuration or traffic monitoringinformation. Finally, the input device interface 130 may provideauxiliary mouse and keyboard ports 132, 134, and may also support aninternal local geometric pointing input device as described below.

Particular groupings of the aforementioned elements may be packaged invarious manners so as to match particular deployment settings. Forexample, selected element groupings may reside within or upon aperipheral box package, computer-bus-compatible card, or housing 150,where such element groupings may include various A/V transducers. Thenature of the selected package 150, and the manner in which theaforementioned elements are incorporated therein or thereupon asintegrated, modular, plug-in, and/or other types of components, isdependent upon the manner in which the present invention is employed,and may be subject to or adaptive to evolving market forces and embeddedlegacy equipment investments. Three exemplary types of packages aredescribed in detail hereafter.

FIG. 2 is a high-level perspective view illustrating a box package 160for the multimedia collaboration device 100. This illustrative boxpackage 160 comprises a housing 162 having a control panel 164 and acable panel 182. The control panel 164 includes an audio mode control166; a microphone/speaker/headset selector 168; a microphone mutecontrol 170; a hold/resume control 172; AUX video and audio inputs 174,176; a telephone add/remove control 178; and a speaker/earphone volumecontrol 180. The audio mode control 166 facilitates user-selectionbetween stereo microphone and synthetic aperture microphone operation,as further described below. The microphone/speaker/headset selector 168provides for user-selection of different audio input/output interfaces,and the microphone mute control 170 facilitates user control over audioinput muting. The hold/resume control 172 pauses or resumes audio inputsin response to user-selection. The AUX video and audio inputs 174, 176respectively facilitate video and audio input from various sources. Thetelephone add/remove control 178 provides control of the insertion of anoptional bridge or coupling to a telephone line for two-way audiocontact with an addition of third-party telephone user. The supportingelectrical couplings would provide for standard telephone loop-through.In one embodiment, the telephone add/remove control 178 includesconventional telephone line echo cancellation circuitry to remove theundesired transmit/receive coupling effects introduced by telephoneloops. Finally, the speaker/earphone volume control 180 controls theamplitude of an audio signal delivered to speakers or an earphone (inaccordance with the setting of the microphone/speaker/headset selector168). Some implementations may include separate volume controls forspeakers, earphones, and/or auxiliary audio I/O.

The cable panel 182 on the box package 160 includes inputs and outputsthat facilitate coupling to a camera/microphone cable 184; a premisesUTP cable 186; left and right speaker cables 188, 190; a video monitoror video overlay card cable 192; and a UTP computer networking cable194.

The box package 160 is suitable for use with a companion desktop orportable computer, and could reside, for example, underneath, atop, oradjacent to a computer or video monitor. Furthermore, a single boxpackage 160 may be used to provide a plurality of companion computers 50with multimedia collaboration capabilities, for example, in a smalloffice environment.

Those skilled in the art will understand that the above combination offeatures is illustrative and can be readily altered. Those skilled inthe art will also understand that in an alternate embodiment, the boxpackage 160 could include a built-in microphone or microphone array, aswell as one or more speakers. Furthermore, those skilled in the art willunderstand that one or more controls described above could beimplemented via software.

FIG. 3 is a suggestive high-level drawing showing the format of aplug-in card package 200 for the multimedia collaboration device 100.The plug-in card package 200 comprises a circuit board or card 202having a standard interface 204 that facilitates insertion into anavailable slot within a computer. For example, the standard interface204 could comprise plated connectors that form a male PeripheralComponent Interface (PCI) connector, for insertion into a female PCIslot coupled to a PCI bus. The elements comprising the multimediacollaboration device 100 may be disposed upon the card 202 in the formof discrete circuitry, chips, chipsets, and/or multichip modules. Thecard 202 includes inputs and outputs for coupling to a camera/microphonecable 214; left and right speaker cables 206, 208; a premises UTP cable210; and a UTP-to-computer cable 212 that facilitates pass-through ofdata networking signals to an existing data networking card. It isunderstood that conventional PCI bus interface electronics and firmwaremay be added to this configuration. Alternatively, the PCI bus maysimply be used to provide power and electrical reference grounding.

The multimedia collaboration device 100 may include more extensive datanetworking capabilities, capable in fact of supporting essentially allthe networking needs of one or more companion or host computers, asdescribed in detail below. In this variation, the plug-in card package200 may therefore be used to provide a computer into which it isinserted with complete data networking capabilities in addition tomultimedia collaboration capabilities via transfer of data networkingpackets between the interface 204 and the computer, in which case theUTP-to-computer cable 212 may not be necessary. The presence of theplug-in-card package 200 may therefore obviate the need for a separatenetwork interface card (NIC) in market situations in which sufficientevolution stability in data networking technologies exists.

The plug-in card package 200 may be used to provide older orless-capable computers with comprehensive, up-to-date real-timemultimedia collaboration capabilities. Alternatively, the plug-in cardpackage 200 can provide video overlay multimedia capabilities tocomputer systems having a monitor for which a video overlay card isunavailable or difficult to obtain. In the event that video overlaymultimedia capabilities are to be delivered to a display or videomonitor other than that utilized by the companion computer 50, theplug-in card package 200 may include a port that facilitates coupling ofa video monitor or video overlay card cable 192 in a manner analogous tothat shown in FIG. 2. A host computer 50 that incorporates a pluralityof plug-in card packages 200 could be used as a multimedia collaborationserver for other computers, in a manner understood by those skilled inthe art.

Those skilled in the art will additionally understand that one or moreof the physical panel controls described above with reference to the boxpackage 160 would be implemented via software control for the plug-incard package 200.

FIG. 4 is a perspective view of a stand-alone package 300 for themultimedia collaboration device 100 that includes a range ofadvantageous internal A/V transducer configurations. In one deployment,the stand-alone package may be attached, mounted, or placed proximate tothe side of a computer monitor or laptop/palmtop computer, and hence isreferred to herein as a “side-kick” package 300.

The side-kick package 300 provides users with a self-containedhighly-localized multimedia communication interface. The incorporationof the microphone array 304 into the side-kick package 300 assists incontrolling the present invention's superior audio performance relativeto adaptive echo-canceled stereo microphone and adaptive echo-canceledmono synthetic aperture microphone capabilities described below. Theplacement of the camera 306 in close proximity to the flat displaydevice 312 aids in maintaining good user eye contact with a displayedimage, which in turn better simulates natural person-to-personinteractions during videoconferencing. The eye contact can be furtherimproved, and manufacturing further simplified, by an integratedcamera/display device as described below with reference to FIGS. 16through 25.

The side-kick package 300 can be used in conjunction with a companioncomputer 50, or in a stand-alone manner. When used with a companioncomputer 50, the side-kick package 300 eliminates the need to consumecompanion computer screen space with a video window. As a stand-alonedevice, the side-kick package 300 can be used, for example, in officereception areas; public kiosks; outside doorways; or alongsidespecial-purpose equipment for which explicatory, possibly interactiveassistance may be useful, such as a photocopier.

Relative to FIG. 2, like reference numbers designate like elements. Theside-kick package 300 comprises a housing 302 in which the multimediacollaboration device 100 described above plus additional elements suchas an internal shock-mounted microphone array 304; a camera 306 that mayinclude auto-focus, auto-iris, and/or electronic-zoom features;acoustically-isolated stereo speakers 308; a thumbstick mouse or similartype of geometric input device 310; and a flat display device 312 mayreside. The side-kick package 300 may further include display brightnessand contrast controls 314, 316, and/or one or more auxiliary audio levelcontrols 180. Additionally, the side-kick package 300 may include acontrol panel having physical panel controls such as an audio modecontrol 166; a microphone/speaker/headphone selector 168; a microphonemute control 170; a hold/resume control 172; AUX video and audio inputs174, 176; and a telephone add/remove control 178, which function in themanner previously described. Those skilled in the art will understandthat the functions of one or more of the physical controls shown in FIG.4 could be implemented so as to be controlled remotely via software. Insome arrangements, there might not be any physical controls, in whichcase control is facilitated by GUIs executing on one or more companioncomputers 50. Ideally, this embodiment may include both physical andremote software controls so that it can operate as a fully stand-alonedevice as well as a slave device supporting applications running on thecompanion computer 50.

The side-kick package 300 has ports for coupling to a premises UTP cable336 and an optional UTP-to-computer cable 338. The side-kick package 300may also include another connector set 334, which, for example,facilitates coupling to a headset, an auxiliary mouse, and/or anauxiliary keyboard. FIG. 4 additionally depicts an overlay window 340upon the flat display device 312, which may be realized via graphicsoverlay capabilities. The graphics overlay capabilities can implementmenus or windows 340 that can provide a user with information such astext or graphics and which may be selectable via the input device 310,creating internal stand-alone GUI capabilities.

Relative to each package 160, 200, 300 described herein, use of themultimedia collaboration device 100 with one or more companion computers50 to effect digital networked A/V communication advantageously spareseach companion computer 50 the immense computational and networkingburdens associated with transceiving and encoding/decoding A/V streamsassociated with A/V capture and presentation. The invention may alsoincorporate additional video graphics features in any of the packages160, 200, 300 described above, such as telepointing over live videoand/or video frame grab for transference to or from a companion or hostcomputer 50.

While FIG. 1 provides a broad overview of the architecture of thepresent invention, specific architectural details and variousembodiments are elaborated upon hereafter, particularly with referenceto FIGS. 5, 6, and 7.

4.3 Architectural Details

FIG. 5 is a block diagram of a first embodiment of a multimediacollaboration device 10 constructed in accordance with the presentinvention, and which provides primary and auxiliary (AUX) support foranalog A/V, and further provides support for networked digitalstreaming. With reference to FIG. 1, like reference numbers designatelike elements. The embodiment shown in FIG. 5 supports analog A/V, andcomprises the preamplifier and buffer unit 102; the audio signalconditioning unit 104; the A/V switch 106; the analog A/V UTPtransceiver 108; the pair splitter 110; a first and a second digitaltransceiver 111, 135; the routing unit 112; a network interface unit114; an analog-to-digital (A/D) and digital-to-analog (D/A) converter116 a; an A/V compression/decompression (codec) unit 116 b; at leastone, and possibly multiple, processors 118.1, 118.n; the memory 120; theI/O interface 130; and the companion and premises network ports 136,138. An internal bus 115 couples the network interface unit 114, the A/Vcodec 116 b, each processor 118.1, 118.n, the memory 120, and the I/Ointerface 130. Each of the audio signal conditioning unit 104, the A/Vswitch 106, the analog A/V UTP transceiver 108, the routing unit 112,and the A/D-D/A converter 116 a may also be coupled to the internal bus115, such that they may receive control signals from the processors118.1, 118.n.

The preamplifier and buffer unit 102 is coupled to receive left andright microphone signals from a left and right microphone 140.1, 140.2,respectively; and a camera signal from the camera 142. It is understoodthat additional microphones 140.3 . . . 140.x and processing 118 and/orswitching capabilities 106 may be included to enhance the syntheticaperture microphone capabilities described below. The preamplifier andbuffer unit 102 may further receive AUX A/V input signals from one ormore auxiliary A/V input devices such as an external VCR, camcorder, orother device. The preamplifier and buffer unit 102 respectively outputsleft and right speaker signals to a left and a right speaker 144.1,144.2; and a display signal to the display device 146. The preamplifierand buffer unit 102 may also deliver AUX A/V output signals to one ormore auxiliary devices.

The audio signal conditioning unit 104 facilitates the adjustment ofoutgoing audio signal volume in conjunction with providing adaptive echocancelled stereo microphone or mono synthetic aperture microphoneprocessing operations upon audio signals received from the preamplifierand buffer unit 102. FIG. 8 is a block diagram of an adaptiveecho-canceled stereo microphone unit 103 within the audio signalconditioning unit 104. The adaptive echo-canceled stereo microphone unit103 comprises a stereo echo canceller 310 and a stereo volume controlunit 350.

The stereo echo canceller 310 comprises conventional monaural echocanceller subsystems that function in a straightforward manner readilyapparent to those skilled in the art. This arrangement includes a leftmicrophone/left speaker (LM/LS) adaptive acoustic echo filter model 312;a left microphone/right speaker (LM/RS) adaptive acoustic echo filtermodel 314; a right microphone/left speaker (RM/LS) adaptive acousticecho filter model 316; and a right microphone/right speaker (RM/RS)adaptive acoustic echo filter model 318. It will be readily understoodby those skilled in the art that linear superposition results in stereoecho canceling capabilities for stereo microphones and stereo speakers.

The stereo volume control unit 350 is coupled to a volume adjustmentcontrol such as described above with reference to the various packageembodiments 160, 200, 300 shown in FIGS. 2, 3, and 4, and is furthercoupled to receive the left and right speaker signals. The stereo volumecontrol unit 350 is also coupled to each model 312, 314, 316, 318 inorder to maximize the utilization of DSP arithmetic and dynamic rangethroughout the full range of speaker volume settings. It is understoodthat stereo balance controls can be implemented using the same stereovolume control elements operating in complimentary increments.

The LM/LS and LM/RS models 312, 314 are coupled to receive the left andright speaker signals, respectively. Similarly, the RM/LS and RM/RSmodels 316, 318 are respectively coupled to receive the left and rightspeaker signals 300. Each of the LM/LS, LM/RS, RM/LS, and RM/RS models312, 314, 316, 318 incorporates an adaptive coefficient tapped delayline weighting element coupled to its corresponding microphone 140.1,140.2 and speaker 144.1, 144.2 in a conventional manner. Additionally,the LM/LS and LM/RS models 312, 314 maintain conventional couplings tothe left microphone 140.1 to facilitate initial acoustic environment andsubsequent adaptive acoustic training operations. Similarly, the RM/LSand RM/RS models 316, 318 maintain couplings to the right microphone140.2 to facilitate these types of training operations.

The stereo echo canceller 310 additionally includes a first signalsummer 320 coupled to outputs of the left microphone 140.1, the LM/LSmodel 312, and the LM/RS model 314; plus a second signal summer 322coupled to outputs of the right microphone 140.2, the RM/LS model 316,and the RM/RS model 318. The first signal summer 320 delivers a leftecho-canceled signal to the A/V switch 106, and the second signal summer322 delivers a right echo-canceled signal to the A/V switch 106, in amanner readily understood by those skilled in the art.

In one embodiment, the stereo echo canceller 310 and stereo volumecontrol unit 350 are implemented together via DSP hardware and software.Furthermore, a single DSP may be used to implement the stereo echocanceller 310, the stereo volume control unit 350, and the adaptiveecho-canceled mono synthetic aperture microphone unit 105, which isdescribed below. In an exemplary embodiment, such a DSP may comprise aTexas Instruments TMS320C54x generation processor (Texas InstrumentsIncorporated, Dallas, Tex.).

In the event that a user employs an earphone, headphone set, or AUXaudio device in conjunction with the present invention, as describedabove with reference to the box, card, and side-kick packages 160, 200,300, the stereo echo canceller 310 is placed in a bypassed, inactive, orquiescent state and the DSP and stereo volume control unit 350facilitate normalization and/or volume adjustment in a conventionalmanner as understood by those skilled in the art. Alternatively,separate volume control and/or normalization circuitry could be providedwhen stereo microphones or the stereo echo canceller 310 is not needed.These may be implemented in various ways with respect to the pathsconnecting to the A/V switch.

FIG. 9 is a block diagram of an adaptive echo-canceled mono syntheticaperture microphone unit 105 within the audio signal conditioning unit104. With reference to FIG. 8, like reference numbers designate likeelements. The adaptive echo-canceled mono synthetic aperture microphoneunit 105 comprises the volume control unit 350 plus a synthetic aperturemicrophone processing unit 330, which may include hardware and/orsoftware. The synthetic aperture microphone processing unit 330comprises a synthetic aperture microphone unit 340 which may includehardware and/or software to implement synthetic aperture microphoneprocessing algorithms; a synthetic microphone/left speaker (SM/LS) model332; a synthetic microphone/right speaker (SM/RS) model 334; and asignal summing circuit 336, each coupled in the manner shown.

The synthetic aperture microphone unit 330 is coupled to receive theleft and right microphone signals, and additionally includesconventional adaptive coefficient weighting and training couplings.Taken together, the synthetic aperture microphone unit 330, the leftmicrophone 140.1, and the right microphone 140.2 (plus one or moreadditional microphones that may be present) form a mono-output syntheticaperture microphone. The synthetic aperture microphone unit 330 performsdelay and/or frequency dispersion operations upon the left and rightmicrophone signals to internally create or define an audio receptionsensitivity distribution pattern in a manner readily understood by thoseskilled in the art. The audio reception sensitivity distribution patternincludes one or more spatial regions referred to as “hot-spots,” as wellas a set of spatial regions referred to as “rejection regions.”Typically, a set of one or more “hot-spots” includes a primary hot-spotof maximal audio reception sensitivity that has a particular position ororientation relative to the geometry of the microphone array 140.1,140.2. The rejection regions comprise spatial positions in which thesynthetic aperture microphone has minimal audio reception sensitivity.

FIG. 10 is an illustration showing an exemplary localized primaryhot-spot 10-3 and a surrounding rejection region 10-8. Within theprimary hot-spot 10-3, the synthetic aperture microphone 10-2 can detectsound waves produced by a speaker 10-1. The location of the primaryhot-spot may be adjusted in accordance with particular conditions in anacoustic environment. In one embodiment, the position or orientation ofthe primary hot-spot may be modified under software control. This inturn could facilitate user-directed hot-spot positioning for optimizingaudio performance in different acoustic situations. FIG. 11 is anillustration showing exemplary primary hot-spot directivity, where thesynthetic aperture microphone 11-2 captures directionally-specificspeech energy from a user 11-1 within a primary hot-spot 11-3 that isoffset relative to that shown in FIG. 10. A rejection region 11-8 existsoutside the primary hot-spot 11-3 in a conventional manner.

The synthetic aperture microphone can additionally reject reflectedspeech energy that originated within the primary hot-spot and thatapproaches the microphone array 140.1, 140.2 from angles beyond thosethat span the primary hot-spot. FIG. 12 is an illustration showingexemplary reflected speech energy rejection. The synthetic aperturemicrophone 12-2 detects sound waves produced by a user 12-1 within aprimary hot-spot 12-3. The synthetic aperture microphone 12-2 rejectssound waves 12-5, 12-6 originating within the primary hot-spot 12-3 andreflected from nearby surfaces because the reflected sound waves arelikely to travel through one or more rejection regions 12-8 along theirreflection path.

The synthetic aperture microphone is further advantageous by virtue ofgood ambient acoustical noise rejection performance. FIG. 13 is anillustration showing exemplary ambient audio noise rejection, in which asynthetic aperture microphone 13-2 rejects conversational noise 13-4 andvarious forms of outside or environmental noise 13-5, 13-6, 13-7. Thenoise and noise reflections traveling towards the microphone array140.1, 140.2 enter a rejection region 13-8 through various directions,and hence are strongly attenuated via the synthetic aperturemicrophone's directional rejection behavior. This is in contrast to auser 13-1 within a primary hot-spot 13-3, who produces sound waves thatthe synthetic aperture microphone 13-2 readily detects with highsensitivity.

Referring also now to FIGS. 5 and 9, the synthetic aperture microphoneunit 330 outputs a mono microphone signal having a magnitude that mostdirectly corresponds to the amount of audio energy present within theset of hot-spots, and in particular the primary hot-spot. The syntheticaperture microphone output signal has little contribution from audioenergy entering from any rejection region directions. Those of ordinaryskill in the art will understand that multiple microphones can be usedto extract voice information from background noise that is in factlouder than the actual speech using adaptive cancellation techniquessuch as those described by Boll and Pulsipher in IEEE Transactions onAcoustics, Speech, and Signal Processing, Vol. ASSP-28, No. 6, December1980. This could be incorporated as a third operational mode for theaudio DSP, for supporting extreme noise environments as might be foundon public streets or repair depots, for example.

The volume control unit 350 is coupled to the left and right speakersignals, as are the SM/LS and SM/RS models 332, 334. The signal summingcircuit 336 is coupled to the output of the synthetic aperturemicrophone unit 340, as well as outputs of the SM/LS and SM/RS models332, 334, and delivers an echo-canceled mono synthetic aperturemicrophone signal to the A/V switch 106.

In one embodiment, the adaptive echo-canceled synthetic aperturemicrophone unit 105 comprises DSP hardware and/or software. The presentinvention can thus provide either adaptive echo-canceled stereomicrophone or adaptive echo-canceled mono synthetic aperture microphonecapabilities in response to user selection. In an exemplary embodiment,the adaptive echo-canceled synthetic aperture microphone unit 105 isimplemented in a DSP such as the Texas Instruments TMS320C54x processorreferenced above. Those skilled in the art will recognize that a singleDSP system can be configured to provide both the adaptive echo-canceledstereo and mono synthetic aperture microphone capabilities describedherein as distinct or integrated operating modes.

In the event that a user employs an earphone, headphone set, or AUXaudio devices in conjunction with the present invention, the syntheticaperture microphone unit 330 is placed in a bypassed, inactive, orquiescent state and the DSP and/or volume control unit 350 facilitateconventional normalization and adjustment of output signal amplitude, ina manner understood by those skilled in the art. Alternatively, separatenormalization and/or volume control circuitry could be provided toaccommodate the aforementioned devices.

Referring again to FIG. 5, the A/V switch 106 comprises conventionalanalog switching circuitry that is coupled to the preamplifier andbuffer unit 102, the audio signal conditioning unit 104, the analog A/VUTP transceiver 108, and the A/D-D/A converters 116 a. The A/V switch106 further maintains a coupling to the internal bus 115, therebyfacilitating processor control over A/V switch operation.

The A/V switch 106 routes incoming signals generated by the left andright microphones 140.1, 140.2 (or larger microphone array), the camera142, and/or any AUX A/V input devices to the analog A/V UTP transceiver108 or the A/D-D/A converters 116 a under the direction of a controlsignal received via the internal bus 115. Similarly, the A/V switch 106selects either the analog A/V UTP transceiver 108 or the A/D-D/Aconverters 116 a as a source for outgoing signals directed to the leftand right speakers 144.1, 144.2, the display device 146, and/or any AUXA/V output devices.

The analog A/V UTP transceiver 108 comprises a conventional analog A/Vtransceiver that provides a signal interface to a first set of UTP wiresthat carry analog A/V signals and which couple the analog A/V UTPtransceiver 108 to the pair splitter 110. The pair splitter 110 isfurther coupled to the first digital transceiver 111 via a second set ofUTP wires that carry digital A/V signals. The analog A/V UTP transceiver108 may be reconfigurable, supporting a range of analog 4-pair, 2-pair,or 1-pair signal transmission methodologies. The selection of anyparticular signal transmission methodology may be performed underprocessor control or by physical configuration switching. Similarly,distance compensation adjustments may be performed under processorcontrol or via physical switching, or alternatively through automaticcompensation techniques in a manner understood by those skilled in theart.

The first and second digital transceivers 111, 135 provide conventionaldigital interfaces to UTP wiring, and are coupled to the routing unit112 in the manner shown. The second digital transceiver 135 is furthercoupled to the companion computer port 136. The first and second digitaltransceivers 111, 135 may be implemented using portions of a standardNIC, as described below, or by other means. In addition to theaforementioned couplings, the routing unit 112 is coupled to the networkinterface unit 114. The routing unit 112 comprises conventional networkhub or mini-hub circuitry. In one embodiment, the routing unit 112performs hard-wired signal distribution and merge functions. In analternate embodiment, the routing unit 112 performs data packet deliverypath selection operations.

The network interface unit 114 comprises conventional network interfacecircuitry, for exchanging data with the internal bus 115 and datapackets with either the multimedia network 60 or a companion computer 50via the premises and companion computer network ports 138, 136 inaccordance with a conventional networking protocol. In one embodiment,the network interface unit 114 is implemented as at least one standardNIC. The NIC may typically include built-in data packet addressexamination or screening capabilities, and hence simplify the routingunit's function to one of communications distribution and mergefunctions in such an embodiment. These distribution and merge functionsserve to provide simultaneous signal or packet exchange among each ofthe premises network port 138, the NIC 114, and the companion computerport 136. One advantage of an embodiment employing a standard NIC isthat the NIC could be easily replaced or upgraded to accommodatetechnological evolution. This range of possibilities is further enhancedby the switching arrangement described below with reference to FIG. 15.Although not shown, it is again understood that should ISDN support bedeemed valuable, network connectors, interface electronics, inversemultiplexers, and Q.931 call control can be introduced through, forexample, connection to the internal bus 115 in a manner familiar tothose skilled in the art.

Taken together, the premises network port 138, the pair splitter 110,the analog A/V UTP transceiver 108, the digital transceiver 111, therouting unit 112, the network interface unit 114, and the companioncomputer port 136 form 1) a first multimedia network interface forhandling analog A/V signals; and 2) a second multimedia networkinterface for handling digital A/V and data networking signals. FIG. 14is a block diagram of a first embodiment of a first 400 and a second 410multimedia network interface provided by the present invention. Thefirst multimedia network interface 400 comprises the aforementionedfirst set of UTP wires plus the analog A/V UTP transceiver 108. Thefirst multimedia network interface 400 facilitates the exchange ofanalog A/V signals between the premises network port 138 and the analogA/V UTP transceiver 108. The second multimedia network interface 410comprises the second set of UTP wires, the digital transceiver 111, therouting unit 112, and the network interface unit 114, which are coupledin the manner shown. In some implementations, the digital transceiver135 may also be a NIC that may be either similar to or different from aNIC employed in the network interface unit 114. The second multimediainterface 410 facilitates the exchange of digital A/V and datanetworking signals between the premises network port 138, the networkinterface unit 114, and the companion computer port 136.

FIG. 15 is a block diagram of a second embodiment of first and secondmultimedia network interfaces provided by the present invention. Thefirst and second multimedia network interfaces are implemented via apassive switching arrangement and/or an active analog switching matrix420 that includes low-capacity, high-frequency analog protectiondevices. Such protection devices may comprise three-terminal,back-to-back diode arrangements, as employed in a Motorola BAV99LT1(Motorola, Inc., Schaumberg, Ill.). In this arrangement, the analogtransceiver 108 may support a number of 4, 2, and 1 pair formats, whichmay be dictated by the marketplace. Alternatively, the analogtransceiver 108 can be a replaceable module.

In the event that data networking is implemented via Gigabit Ethernet orother network protocol that conventionally consumes the entire physicalcapacity of an entire RJ-45 connector, the present invention may employan additional RJ-45 or other type of connector for carrying analog A/Vsignals.

Via the second multimedia network interface, the present inventionprovides internal data communication transmit, receive, and routingcapabilities. An external or companion computer 50 can therefore issuecontrol signals directed to the present invention in accordance withstandard data networking protocols. The second multimedia networkinterface can also provide “loop-through” signal routing between thepremises network port 138 and the companion computer port 136.Additionally, the data routing capabilities provided by the secondmultimedia network interface facilitate coupling to both existingbroadcast or switching hubs. The second multimedia network interfacealso supports the transfer of digital A/V streams. Thus, the secondmultimedia network interface cleanly separates data communicationsdirected to one or more companion computers 50, the multimedia network60, and the multimedia collaboration device 10.

Once again referring to FIG. 5, each of the A/V switch 106, the analogA/V UTP transceiver 108, the routing unit 112, the network interfaceunit 114, the A/V codec 116 b, the set of processors 118.1, 118.n, thememory 120, and the I/O interface 130 is coupled to the internal bus115. The A/V codec 116 b is further coupled to the A/D-D/A converters116 a, which are coupled to the A/V switch 106. It is noted that theA/D-D/A converters 116 a may include color-space conversion capabilitiesto transform between RGB and YUV or other advantageous color spaces.

The memory 120 comprises Random Access Memory (RAM) and Read-Only Memory(ROM), and stores operating system and application software 122, 124.Depending upon the nature of the processors 118.1, 118.n, the operatingsystem 122 could comprise a scaled-down, conventional, or enhancedversion of commercially-available operating system software, and/orspecial-purpose software. In an exemplary embodiment, the operatingsystem 122 comprises Windows CE (Microsoft Corporation, Redmond, Wash.)or another commercial product selected in accordance with the particularenvironment in which the present invention is employed. The applicationsoftware 124 may comprise programs for performing videoconferencing,messaging, publishing, broadcast reception, and media-on-demandoperations, and internet browsing using programs such as NetscapeNavigator (Netscape Communications Corporation, Mountain View, Calif.).Depending upon the nature of the processors 118.1, 118.n, the internetbrowser program could be a scaled down, conventional, or augmentedversion of a commercially-available browser.

The processors 118.1, 118.n manage communication with the networkinterface unit 114, and control the overall operation of the multimediacollaboration device 10 in accordance with control signals received viathe network interface unit 114. The processors 118.1, 118.n additionallyprovide graphics overlay capabilities, and may further provide internetbrowsing capabilities in conjunction with application software 124 aspreviously described. Relative to managing communication with thenetwork interface unit 114, the processors 118.1, 118.n may manageprotocol stacks and/or state machines. With regard to controlling theoverall operation of the multimedia collaboration device 10, theprocessors 118.1, 118.n issue control signals to the A/V switch 106 andexecute application software resident within the memory 120. Thegraphics overlay capabilities facilitate the placement of fonts,cursors, and/or graphics over video present upon the display device 146.With sufficient processing power, the present invention can serve as astand-alone, real-time video-capable internet appliance.

As described above, the A/D-D/A converters 116 a may compriseconventional circuitry to perform color-space conversion operations inaddition to analog-to-digital and digital-to-analog signal conversion.The A/V codec 116 b comprises conventional A/V signal encoding anddecoding circuitry, and provides the present invention with compressionand decompression capabilities. Together these enable the presentinvention to encode and decode A/V streams without loading down acompanion computer's processing and networking power. Either of thefirst or second multimedia network interfaces described above can routedigital A/V signals to the A/V codec 116 b, while routing non-A/Vsignals to the companion computer 50. The present invention's ability toencode and decode A/V signals independent of a companion or externalcomputer is particularly advantageous in situations in which videosignal encoding and decoding must occur simultaneously, such as in 2-wayteleconferencing or network-based video editing applications. Thepresent invention may support network-based video editing applicationsbased upon a high bandwidth near-zero-latency compression approach,which can be implemented, for example, through JPEG or waveletcompression operations; or an interim compression approach.

In one embodiment, the A/V codec 116 b comprises a chip or chipset. Inanother embodiment, the A/V codec 116 b comprises a processor 118.kcapable of performing compression and decompression operations. In moreadvanced implementations, the A/V codec 116 b could comprise a singleprocessor 118.m capable of performing user interface functions inaddition to A/V compression and decompression operations. Such animplementation could also provide an Application Program Interface (API)in conjunction with operating system software 122. In an exemplaryembodiment of such an implementation, the A/V codec 116 b may comprise aNUON processor (VM Labs, Mountain View, Calif.).

4.4 Additional Embodiments

FIG. 6 is a block diagram of a second embodiment of a multimediacollaboration device 20, which provides primary support for analog audioI/O and digital visual I/O, and further supports analog and digitalauxiliary A/V I/O, plus networked digital streaming. Relative to FIG. 5,like reference numbers designate like elements.

The second embodiment of the multimedia collaboration device 20 includesa digital camera 152, a digital display device 154, a digital AUX A/Vinterface 156, and a stream selector 158. The digital camera 152 and thedigital display device 154 respectively capture and display images in aconventional manner. The digital AUX A/V interface 156 facilitatesbidirectional coupling to auxiliary digital A/V devices, such as anexternal computer, a digital VCR, or Digital Versatile Disk (DVD)player. Each of the digital camera 152, the digital display device 154,and the digital AUX A/V interface 156 is coupled to the stream selector158, which is coupled to the A/V codec 116 b.

The stream selector 158 comprises conventional circuitry thatselectively routes digital streams between the A/V codec 116 b and thedigital camera 152, the digital display device 154, and the digital AUXA/V interface 156. The stream selector 158 may route incoming digitalimage streams received from either of the digital camera 152 or thedigital AUX A/V interface 156 to the A/V codec 116 b. In one embodiment,the stream selector 158 may be capable of multiplexing between these twoincoming digital stream sources. Undersampling may also be used tofacilitate the compositing of multiple video images. Relative tooutgoing digital image streams, the stream selector 158 may route suchstreams to either or both of the digital display device 154 and digitalAUX A/V interface 156, where such routing may occur in a simultaneous ormultiplexed manner. The stream selector 158 additionally facilitates theexchange of digital audio streams between the A/V codec 116 b and thedigital AUX A/V interface 156.

The A/V codec 116 b and the A/D-D/A converters 116 a together facilitatethe conversion of digital A/V signals associated with the digital camera152, the digital display device 154, and/or auxiliary digital A/Vdevices into analog A/V signals. The A/V switch 106 facilitates exchangeof these analog A/V signals with AUX A/V devices and/or the premisesnetwork port 138.

Because the A/V codec 116 b is also coupled to the internal bus 115 andhence to the network interface unit 114, digital A/V signals capturedfrom the digital camera 152 or directed to the digital display 154 orreceived from the digital AUX A/V interface 156 may be packetized andexchanged via the premises network port 138 and/or the companioncomputer port 136.

FIG. 7 is a block diagram of a third embodiment of a multimediacollaboration device 30, which provides primary support for analog audioI/O and digital visual I/O, support for digital auxiliary A/V I/O, andsupport for networked digital streaming. Relative to FIGS. 5 and 6, likereference numbers designate like elements.

The third embodiment of the multimedia collaboration device 30 includesa digital camera 152, a digital display device 154, a digital AUX A/Vinterface 156, and a stream selector 158 in the manner described above.Analog audio signals associated with the microphones 140.1, 140.2 andspeakers 144.1, 144.2 are routed through the A/D-D/A converters 116 aand A/V codec unit 116 b. Thus, the third embodiment of the presentinvention manages digital A/V streams, and may exchange such streamswith the multimedia network 60 and/or a companion computer 50. The thirdembodiment of the multimedia collaboration device 30 does not transmitanalog A/V signals over the multimedia network 60, and hence the analogswitching unit 106, the analog A/V UTP transceiver 108, and the pairsplitter 110 described above relative to the first and second multimediacollaboration device embodiments are not required.

4.5 Camera and Display Device Integration

As previously indicated, placement of the camera 142 in close proximityto the display device 146 aids in maintaining good user eye-contact witha displayed image, thereby closely approximating natural face-to-facecommunication in videoconferencing situations. Essentially perfecteye-contact can be achieved by integrating a large-area photosensorarray with a large-area array of emissive or transmissive devices thatform the basis for display device pixels.

Multiple photosensor and display element integration techniques exist.In general, the formation of an image using a photosensor arraynecessitates the use of optical elements in conjunction with photosensorelements. Photosensor and display element integration techniques aredescribed in detail hereafter, followed by image formationconsiderations relative to integrated photosensor/display elementarrays.

4.6 Display Pixel and Photosensor Element Interleaving

One way of integrating photosensor elements with emissive ortransmissive display elements is via element interleaving. FIG. 16 is anillustration showing a first photosensor and display elementinterleaving technique, in which display elements 510 and photosensorelements 520 populate a viewing screen 502 in an alternating manner.Each display element 510 generates or transmits light corresponding to aparticular color or set of colors. Similarly, each photosensor element520 detects light corresponding to a particular color. As described indetail below, display elements 510 and photosensor elements 520 operatein a temporally and/or spatially separated manner relative to each otherto ensure that image capture is essentially unaffected by image display.

Display and photosensor elements 510, 520 corresponding to a particularcolor are interleaved in accordance with a color distribution scheme.FIG. 17 is an illustration of an exemplary photosensor element color anddisplay element color distribution scheme. In FIG. 17, display elements510 corresponding to the colors red, green, and blue are identified viathe uppercase letters R, G, and B, respectively. Photosensor elements520 corresponding to red, green, and blue are respectively identified bythe lowercase letters r, g, and b. Display elements 510 corresponding toa particular color are offset relative to each other, and interleavedwith display and photosensor elements 510, 520 corresponding to othercolors. Similarly, photosensor elements 520 corresponding to aparticular color are offset relative to each other, and interleaved withdisplay and photosensor elements 510, 520 corresponding to other colors.Those skilled in the art will recognize that a variety of photosensorand display element color distribution schemes are possible.

The presence of photosensor elements 520 interleaved with displayelements 510 reduces image resolution, and increases pixel pitch (i.e.,the spacing between pixels). To minimize the effect that the photosensorelements 520 have upon the appearance of a displayed image, photosensorelements 520 having or consuming a smaller area than the displayelements 510 are employed. Furthermore, various display and photosensorelement layout geometries may be used to produce an interleaving patternthat closely approximates display element pitch found in conventionaldisplay devices. FIG. 18 is an illustration of a second photosensor anddisplay element interleaving technique, in which photosensor and displayelement geometries and size differentials aid in minimizing pixel pitchand maximizing displayed image resolution. Since a viewer's eye willintegrate or average the light output by groups of display elements 510,interleaving techniques of the type shown in FIG. 18 ensure that theviewer will perceive a high-quality image. Those skilled in the art willunderstand that various microoptic structures or elements, such asmicrolenses, could be employed in the nonluminent spaces between displayelements 510 and/or photosensor elements 520 to reduce or minimize aviewer's perception of nonluminent areas in a displayed image. Suchmicrooptic structures are elaborated upon below.

The display elements 510 referred to herein may comprise essentially anytype of conventional light emitting or transmitting device, such as aLight Emitting Diode (LED) or Liquid Crystal Display (LCD) pixelelement. Similarly, the photosensor elements 520 may compriseessentially any type of conventional light sensing or detecting device.For example, the photosensor elements 520 could comprise photodiodes,such as Schottky or p-i-n photodiodes; phototransistors; capacitive orcharge-coupled devices (CCDs); charge modulated devices (CMDs); or othertypes of light-sensitive devices. The photosensor elements 520 could befabricated, for example, using standard semiconductor processingtechniques employed during the manufacture of flat panel displays.

In a typical display device, a single display element 510 is used tooutput light of a particular color. Display elements 510 based uponorganic electroluminescence are capable of simultaneously generatinglight comprising multiple wavelengths in the visible spectrum, and formthe basis for full-color LED arrays. In particular, a single StackedOrganic Light Emitting Diode (SOLED) pixel element can produce red,green, and blue light. The intensity of each color is independentlytunable, as is each color's mean wavelength. Thus, a single SOLED canform a full-color pixel. As an alternative to organic electroluminescentmaterials, the present invention may employ other full-color transparentor semitransparent luminescent materials, such as light-emitting and/orlight-responsive polymer films.

FIG. 19 is a cross-sectional view showing a full-color pixel arrayintegrated with a photosensor element array upon a common substrate 702such as glass or plastic. As an example, a SOLED 710 is considered asthe full-color pixel technology in the discussion that follows. Thoseskilled in the art will understand that the concepts described hereincan be applied to other full-color pixel technologies. Each SOLED 710comprises a first, second, and third semitransparent electrode 712, 714,716; a first, second, and third organic electroluminescent layer 722,724, 726; and a reflecting contact layer 730, in a manner understood bythose skilled in the art. Each electroluminescent layer 722, 724, 726emits light in a particular wavelength range in response to an appliedelectric field. For example, the first, second, and third organicelectroluminescent layers 722, 724, 726 could respectively output blue,green, and red light.

A color filter 750, an optional microoptic structure 760, and aphotosensor element 520 form a color-specific photosensor element 770that is fabricated adjacent to each SOLED 710. The microoptic 760 maycomprise one or more microlenses, apertures, and/or other types ofplanar optic structures, and serves to focus incoming light onto thephotosensor element 520 to aid image formation in the manner describedbelow. The microoptic structure 760 may be formed through theapplication of conventional microlens or planar optic fabricationtechniques during photosensor element fabrication steps. For example,the microoptic structure 760 may be formed by depositing aselectively-doped dielectric or dielectric stack prior to or duringphotosensor element fabrication, in a manner well understood by thoseskilled in the art.

The color-specific photosensor element 770 may also include one or moreantireflection layers, which are deposited in a conventional manner.Additionally, one or more types of passivation or isolation materials,such as Silicon Dioxide, Silicon Nitride, Polyimide, or spin-on-glassmay be deposited in between each SOLED 710 and color-specificphotosensor element 770 in a manner understood by those skilled in theart.

Each color-specific photosensor element 770 detects light characterizedby a specific wavelength interval. Thus, while any given SOLED 710 maysimultaneously output red, green, and/or blue light, separatecolor-specific photosensor elements 770 are used to individually detectred, green, and blue light. Because each SOLED 710 forms a full-colorpixel, integration of a SOLED array with a photosensor array in themanner shown in FIG. 19 is particularly advantageous relative toproviding a high-resolution display having image capture capabilities.

4.7 Display and Photosensor Element Stacking

a) Integrated SOLED/Photosensor Element

A full-color pixel element such as a SOLED 710 and a color-specificphotosensor element 770 can be integrated together, such that theincorporation of a photosensor element array into a display elementarray can be accomplished essentially without a resolution or pixelpitch penalty. FIG. 20 is a cross-sectional view showing an integratedfull-color pixel/photosensor element 800, which may form the basis of anintegrated display element/photosensor element array. For purpose ofexample, the full-color pixel element is considered to be a SOLED 810 inthe description that follows. Those skilled in the art will understandthat other types of full-color pixel technologies could be used toproduce the integrated full-color pixel/photosensor element 800described hereafter.

Relative to FIG. 19, like reference numbers designate like elements. Thefull-color pixel/photosensor element 800 comprises a SOLED 810 having acolor-specific photosensor element 770 fabricated thereupon. Thefull-color pixel/photosensor element 800 is fabricated upon a substrate702 such as glass. The SOLED 810 comprises a first, a second, a third,and a fourth semitransparent electrode 712, 714, 716, 812; a first,second, and third organic electroluminescent layer 722, 724, 726; and apatterned reflecting contact layer 830.

With the exception of the fourth semitransparent electrode 812 and thepatterned reflecting contact layer 830, the SOLED 810 shown in FIG. 20is essentially the same as that depicted in FIG. 19. The fourthsemitransparent electrode 812 serves as one of the electrodes for thephotosensor element 520 within the color-specific photosensor element770, in a manner readily understood by those skilled in the art.Deposition of the fourth semitransparent electrode 812 may not berequired under the patterned reflecting contact layer 830, and as suchthe SOLED 810 and color-specific photosensor element 770 may not share acommon electrical interface layer. The patterned reflecting contactlayer 830 comprises conventional contact materials or metals that havebeen patterned to include a gap or opening.

The color-specific photosensor element 770 is fabricated on top of thefourth semitransparent electrode 812, in the opening defined in thepatterned reflecting contact layer 830. The color-specific photosensorelement 770 thus detects light that has been transmitted through thesubstrate 702 and each of the first through fourth semitransparentelectrodes 712, 714, 716, 812. Those skilled in the art will understandthat the location of the opening defined in the patterned reflectingcontact layer 830, and hence the location of the color-specificphotosensor element 770 upon the SOLED 810, may vary among adjacentfull-color pixel/photosensor elements to ensure that the human observerperceives a high-quality displayed image. The SOLED 810 and thecolor-specific photosensor element 770 may operate in atemporally-separated manner to ensure that image capture is essentiallyunaffected by image display, as further elaborated upon below.

b) Stacked Full-Color Emitter/Full-Color Detector Structures

A full-color pixel element, such as a stacked organic electroluminescent(SOE) structure, may also be used to detect light. Thus, a singlestructure based upon full-color materials technology may be used forboth RGB light emission and RGB light detection, thereby advantageouslyfacilitating the integration of a photosensor element array and adisplay element array while maintaining small pixel pitch and high imageresolution.

FIG. 21 is a cross-sectional view of a first full-color emitter/detector900. In the description that follows, the first full-coloremitter/detector 900 is considered to be an SOE-based device. Thoseskilled in the art will recognize that other full-color technologiescould be employed to produce the first full-color emitter/detector 900in alternate embodiments.

Relative to FIGS. 19 and 20, like reference numbers designate likeelements. The first full-color emitter/detector 900 is fabricated upon asubstrate 702 such as glass, and comprises first through seventhsemitransparent electrodes 712, 714, 716, 812, 912, 914, 916; firstthrough sixth organic electroluminescent layers 722, 724, 726, 922, 924,926; an optional microoptic structure 920; and a reflecting contactlayer 730.

In the first full-color emitter/detector 900, the first through thirdorganic electroluminescent layers 722, 724, 726 serve as RGB lightemitters controlled by voltages applied to the first through fourthsemitransparent electrodes 712, 714, 716, 812, and thus form a SOLED902. The microoptic structure 920 comprises one or more microlenses,apertures, and/or other planar microoptic structures that focus incominglight into the fourth, fifth, and sixth organic electroluminescentlayers 922, 924, 926, which in turn produces or induces pairwise voltagedifferences across the fifth, sixth, and seventh semitransparentelectrodes 912, 914, 916 and the reflecting contact layer 730. Themicrooptic structure 920, the fourth through sixth organicelectro-luminescent layers 922, 924, 926, the fifth through seventhsemitransparent electrodes 912, 914, 916, and the reflecting contactlayer 730 therefore form a first SOE photosensor 904 for detecting RGBlight.

Light emitted by the SOLED 902 may travel through the substrate 702toward a viewer, or through the first SOE photosensor 904, where it isreflected back toward the substrate 702 by the reflecting contact layer730. The first SOE photosensor 904 detects incoming light that hastraveled through the substrate 702 and the SOLED 902. As described indetail below, SOLED light emission and SOE photosensor light detectionmay occur in a temporally and/or spatially separated manner, such thatimage capture is essentially unaffected by image display.

Those skilled in the art will recognize that the SOLED 902 and the firstSOE photosensor 904 may be able to share a single semitransparentelectrode at their interface in an alternate embodiment (i.e., the firstfull-color emitter/detector 900 may be fabricated without one of thefourth or fifth semitransparent electrodes 812, 912) since SOLED and SOEphotosensor operation within a single first full-color emitter/detector900 may be temporally separated). Those skilled in the art will alsounderstand that in addition to the layers described above, the firstfull-color emitter/detector 900 may include additional microoptic layersand/or one or more antireflective layers. Those skilled in the art willfurther recognize that in an alternate embodiment, the first full-coloremitter/detector 900 could be fabricated such that the first SOEphotosensor 904 resides in contact with the substrate 702, and the SOLED902 resides on top of the first SOE photosensor 904. In such anembodiment, the reflecting contact layer 730 would be incorporated intothe SOLED 902. Those skilled in the art will also recognize that eitheror both of the SOLED 902 and the first SOE photosensor 904 could beimplemented using other types of transparent or semitransparentfull-color device and/or materials technologies in alternateembodiments.

FIG. 22 is a cross-sectional view of a second full-coloremitter/detector 1000. For ease of understanding, the second full-coloremitter/detector is considered to be based upon SOE technology in thefollowing description. Those skilled in the art will recognize thatother full-color materials technologies could be employed to produce thesecond full-color emitter/detector 1000 in alternate embodiments.

Relative to FIG. 20, like reference numbers designate like elements. Thesecond full-color emitter/detector 1000 is fabricated upon a substrate702 such as glass, and comprises a first through fifth semitransparentelectrode 712, 714, 716, 812, 1012; a first through sixth organicelectroluminescent layer 722, 724, 726, 1022, 1024, 1026; an optionalmicrooptic structure 1020; a first, a second, and a third reflectingcontact layer 1032, 1034, 1036; and a first and a second boundarystructure 1042, 1044.

The first through third organic electroluminescent layers 722, 724, 726,in conjunction with the first through fourth semitransparent electrodes712, 714, 716, 812, form a SOLED 902 in a manner analogous to thatdescribed above with reference to FIG. 21. The microoptic structure1020, the first through third organic electroluminescent layers 1022,1024, 1026, the reflecting contact layers 1032, 1034, 1036, and thefirst and second boundary structures 1042, 1044 form a second SOEphotosensor 1004.

Taken together, the fourth, fifth, and sixth organic electroluminescentlayers 1022, 1024, 1026 and the boundary structures 1042, 1042 span anarea essentially equal to that of any semitransparent electrode 712,714, 716, 812, 1012. The first boundary structure 1042 separates thefourth and fifth organic electroluminescent layers 1022, 1024.Similarly, the second boundary structure 1044 separates the fifth andsixth organic electroluminescent layers 1024, 1026. The first, second,and third reflecting contact layers 1032, 1034, 1036 respectively resideupon or atop the fourth, fifth, and sixth organic electroluminescentlayers 1022, 1024, 1026.

The microoptic structure 1020 may comprise one or more microlenses,apertures, and/or other planar microoptic structures that focus incominglight into the fourth, fifth, and sixth organic electroluminescentlayers 1022, 1024, 1026. The fourth organic electroluminescent layer1022 detects incoming photons having a wavelength range associated witha particular color, for example, red. The presence of such photons inthe fourth organic electroluminescent layer produces or induces avoltage difference between the fourth semitransparent electrode 1012 andthe first reflecting contact layer 1032. Similarly, the fifth and sixthorganic electroluminescent layers 1024, 1026 each detect incoming lightcorresponding to a particular wavelength range, for example, green andblue, respectively. The presence of blue and green light respectivelyinduces a voltage difference between the second and third reflectingcontact layers 1034, 1036 and the fourth semitransparent electrode 1012.

Those skilled in the art will recognize that the thickness of each ofthe fourth, fifth, and sixth organic electroluminescent layers 1022,1024, 1026 may be varied in accordance with the particular wavelengthrange that each such layer is to detect. Those skilled in the art willadditionally recognize that the microoptic structure 1020 may befabricated such that its characteristics vary laterally from one organicelectroluminescent layer 1022, 1024, 1026 to another, and that one ormore antireflection layers may be incorporated into the secondfull-color emitter/detector 1000. Moreover, the SOLED 902 and the secondSOE photosensor 1004 may be able to share a single semitransparentelectrode at their interface a manner analogous to that described aboverelative to the first SOE photosensor 904. Finally, those skilled in theart will recognize that either or both of the SOLED 902 and the secondSOE photosensor 1004 could be implemented using other types oftransparent or semitransparent full-color technologies in alternateembodiments.

4.8 Other Integrated Emitter/Detector Structures

As indicated above, a light detecting element may be similar, nearly, oressentially identical in structure and/or composition to a lightemitting element. Because any given emitter/detector structure may beused for light emission during one time interval and light detectionduring another time interval as described below, a single light emittingstructure may also be used for light detection.

FIG. 23 is a cross-sectional diagram of a third full-coloremitter/detector 1100. For ease of understanding, the third full-coloremitter/detector is described hereafter in the context of SOEtechnology. Those skilled in the art will understand that otherfull-color materials and/or technologies could be employed to producethe third full-color emitter/detector 1100 in alternate embodiments.

Relative to FIG. 19, like reference numbers designate like elements. Thethird full-color emitter/detector 1100 is fabricated upon a substrate702 such as glass or plastic. The third full-color emitter/detector 1100comprises a SOLED 710 having a first through a third semitransparentelectrode 712, 714, 716; a first, a second, and a third organicelectroluminescent layer 722, 724, 726; a reflecting top contact layer730. The third full-color emitter/detector 1000 may additionally includea microoptic layer 1120. During a first time interval, the SOLED 710 mayoperate in a light emitting mode in a conventional manner. During asecond time interval, the SOLED 710, in conjunction with the microopticlayer 1120, operates as a photosensor to detect incoming light in amanner analogous to that described above relative to the SOEphotosensors 904.

The microoptic layer 1120 may comprise a microlens and/or other type ofplanar optic structure, and may be fabricated such that differentportions of the microoptic layer 1120 affect light in different manners.This in turn could aid in providing particular light detectionresponsivity while minimally affecting the manner in which light emittedby the third full-color emitter detector 1100 will be perceived by ahuman eye.

FIG. 24 is a top-view of an exemplary microoptic layer 1120 havingdifferent optical regions 1190, 1192 defined therein. A first opticalregion 1190 may allow light to pass in an essentially unaffected manner.A second optical region 1192 serves as a focusing element that producesa desired spatial or modal light intensity pattern within the thirdfull-color emitter/detector. As the second optical region 1192 occupiesa smaller area than the first optical region 1190, its affect upon humanperception of light emitted by the third full-color emitter/detector maybe small or minimal. Those skilled in the art will understand that thelocation of the second optical region 1192 may vary among adjacent thirdfull-color emitter/detectors 1100, to further enhance the quality of adisplayed image seen by a human eye.

In an alternate embodiment, the microoptic layer 1120 could includeadditional optical regions. For example, one or more portions of thefirst optical region 1190 could be designed or fabricated to compensatefor any effects the second optical region 1192 has upon human perceptionof light emitted by the third full-color emitter/detector 1100. Asanother example, the second optical region 1192 could be replaced oraugmented with other, possibly smaller, optical regions distributedacross the plane of the microoptic layer 1120 to further optimize lightdetection and emission characteristics.

4.9 Image Formation

A simple or compound lens is conventionally used to focus an image ontoan array of photosensors. FIG. 25 illustrates a simple or compound lens600 that receives or collects light 602 reflected or emanating from anobject 604, and focuses such light onto a photosensor element array 606.

Relative to a single array that integrates both display and photosensorelements 510, 520, the use of a conventional simple or compound lenswould adversely affect the characteristics of the displayed image. Tofacilitate image detection in such an integrated array, photosensorelements 520 may incorporate microoptic structures and/or apertures, asdescribed above, on an individual basis. Each aperture and/or microopticstructure focuses light received from a small portion of an object ontoa photosensor element 520. As depicted in FIG. 25, sets ofmicrooptic-equipped photosensor elements 520 within a photosensor array620 receive light 622, 624 emanating from different parts of an object626. Those skilled in the art will recognize that the present inventioncould employ microoptic structures or elements that focus light ontomultiple photosensor elements 520 in alternate embodiments, where suchmicrooptic elements may be incorporated onto separates substrates.Signals output by the microoptic-equipped photosensor elements 520 aretransferred to an image processing unit 628 for further processing, asdescribed in detail below.

Conventional display devices comprise multiple rows or lines of displayelements 510, and produce a displayed image on a line-by-line basis.Similarly, conventional photosensor arrays comprise multiple rows ofphotosensor elements 520, which may be scanned on a line-by-line basisduring image capture operations. The integrated displayelement/photosensor element arrays considered herein may also 1) producea displayed image by activating display elements 510 on a line-by-linebasis; and 2) capture light received from an object by detectingphotosensor element output signals on a line-by-line basis.

In one embodiment, the present invention includes a display controlcircuit for performing display line scans that produce a displayed imageon a line-by-line basis, and a capture control circuit for performingphotosensor line scans that read photosensor element output signals on aline-by-line basis. Each of the display and capture control circuitsinclude conventional clocking, address decoding, multiplexing, andregister circuitry. In order to ensure that image capture is essentiallyunaffected by image display (i.e., to prevent light emitted ortransmitted by display elements 510 from affecting incoming lightdetection by adjacent photosensor elements 520), the display line scansand photosensor line scans may be temporally and/or physically separatedrelative to each other. This separation may be controlled viaconventional clocking and/or multiplexing circuitry.

In one embodiment, photosensor line scans are initiated after a displayline scan has generated fifty percent of an image (i.e., after fiftypercent of the display element lines have been activated during a singlefull-screen scan cycle), such that the photosensor line scan trails thedisplay line scan by a number of display element rows equal to one-halfof the total number of display element rows present in the integrateddisplay element/photosensor element array. More generally, the captureline scan could trail the display line scan by a particular timeinterval or a given number of completed display line scans.

In another embodiment, one-half of the display element lines define afirst display field, and one-half of the display element lines define asecond display field, in a manner well understood by those skilled inthe art. Similarly, one-half of the photosensor element lines define afirst photosensor field, and the remaining photosensor element linesdefine a second photosensor field. The first display field and either ofthe first or second photosensor fields may be scanned eithersimultaneously or in a time-separated manner, after which the seconddisplay field and the remaining photosensor field may be scanned in ananalogous manner. Those skilled in the art will recognize that thedisplay and photosensor field scanning can be performed in a manner thatsupports odd and even field scanning as defined for NTSC and PALtelevision standards.

In yet another embodiment, a single full-screen display line scan cycleis completed, after which a single full-screen photosensor line scancycle is completed, after which subsequent full-screen display line andphotosensor line scans are separately performed in a sequential manner.

The set of photosensor element output signals received during any givenphotosensor line scan are transferred to an image processing unit 628.The image processing unit 628 comprises signal processing circuitry,such as a DSP, that performs conventional digital image processingoperations such as two-dimensional overlap deconvolution, decimation,interpolation, and/or other operations upon the signals generated duringeach photosensor line scan. Those skilled in the art will understandthat the number and types of digital image processing operationsperformed upon the signals generated during each photosensor line scanmay be dependent upon the properties of any microoptic structuresassociated with each photosensor element 520. Those skilled in the artwill further understand that signal conditioning circuitry mayadditionally be present to amplify photosensor element signals oreliminate noise associated therewith. Such signal conditioningcircuitry, or a portion thereof, may be integrated with each photosensorelement 520.

The image processing unit 628 forms a conventional final output imagearray using signal processing methods, and outputs image array signalsto a buffer or memory, after which such signals may be compressed andincorporated into data packets and/or converted into analog videosignals for subsequent transmission, where the compression and/orconversion may occur in conjunction with associated audio signals.

The signal processing algorithms employed in image formation aredetermined by the nature of any microoptic elements employed inconjunction with the photosensor elements 520. Such algorithms mayperform deconvolution, edge-effect handling, decimation, and/orinterpolation operations in a manner understood by those skilled in theart.

For example, if the microoptic elements amount to tiny apertures thatlimit detector pixel source light to non-overlapping segments in theprincipal area of view, the signal processing amounts to aggregating thepixels into an array and potentially performing interpolation and/ordecimation operations to match the resolution of the pixel detectorarray to that of the final desired image.

As detection pixels overlap by increasing amounts, the applied signalprocessing operations can advantageously sharpen the image bydeconvolving the impulse response of the pixel overlap function.Depending upon the microoptic arrangement employed, which may bedictated by device cost and fabrication yield or reliability, theoverlap impulse response takes on varying characteristics, affecting thealgorithm the image processing unit 628 is required to perform. Ingeneral, the deconvolution can be handled as either a set oftwo-dimensional iterated difference equations, which are readilyaddressed by standard numerical methods associated with the approximatesolution of differential equations, or through conversions to thefrequency domain and appropriate division operations. Further, if theoverlap function is highly localized, which can be a typical situation,the difference equations can be accurately approximated by neglectinghigher-order terms, which greatly simplifies the resulting operations.This is in contrast to frequency domain techniques for this case, aslocalization in the impulse response implies immense nonlocalization inthe transform domain. However, should the overlap impulse responseitself be far less localized, frequency domain deconvolution methods maybe advantageous. Care must be taken in limiting the division to relevantareas when there are zeros in the frequency-domain representation of theoverlap impulse response (transfer function).

Edge effects at the boundaries of the pixel detector array can behandled by various methods, but if the overlap impulse response is keptlocalized by apertures and/or other microoptic elements, thenundesirable edge effects in the final image formation (that may resultfrom “brute-force” treatment of the edges) quickly vanish within a fewpixels from the boundary of the final formed image. Cropping can then beemployed to avoid such edge effect altogether. Thus, by creating aslightly-oversized pre-final image formation array and eliminating edgeeffect by cropping, a final image array of desired resolution having noedge effects induced by overlap impulse responses can be readilyproduced.

It is known to those skilled in the art that in general, apertureeffects invoked by actual apertures and/or microoptic elements cancreate diffraction patterns or spatial intensity modes in the lighttransmitted through the optical structure. Such optical structures maybe designed to enhance or eliminate particular modes or diffractioneffects, in a manner readily understood by those skilled in the art.

While the teachings presented above have been described in relation to adisplay device having a camera or image capture capabilities integratedtherein or thereupon, the above teachings relating to 1) variousphotosensor element, microoptic and/or apertured structures; and 2)image processing requirements for creating an array of image signalsthat correspond to a captured image can be applied to effectively createa camera disposed or integrated upon any one of a wide variety ofsurfaces or substrates, including glass, plastic, partially-silveredmirrors, or other materials. Photosensor elements 520 disposed upon suchsubstrates may be organized or distributed in a manner similar to thatshown above with reference to FIGS. 16, 17, and 18, with the exceptionthat display elements 510 shown in those figures may not be present.

The principles of the present invention have been discussed herein withreference to certain embodiments thereof. Study of the principlesdisclosed herein will render obvious to those having ordinary skill inthe art certain modifications thereto. The principles of the presentinvention specifically contemplate all such modifications.

1. A display-pixel and photosensor-element device for use as a displayand a camera, the device comprising: a plurality of display elements,each display element comprising a light-emitting diode; and a pluralityof photosensor elements, each photosensor element comprising alight-sensitive device, wherein the plurality of display elements andthe plurality of photosensor elements are fabricated onto an essentiallyplanar surface so as to create a device that is usable as a display anda camera.
 2. The display-pixel and photosensor-element device of claim 1wherein the light-sensitive device comprises a diode that islight-sensitive.
 3. The display-pixel and photosensor-element device ofclaim 2 wherein the diode is configured to act as a photodiode.
 4. Thedisplay-pixel and photosensor-element device of claim 1 wherein thelight-sensitive device is fabricated employing standard semiconductormanufacturing employed during the manufacture of flat panel displays. 5.The display-pixel and photosensor-element device of claim 1 wherein someof the plurality of display elements comprise a light-emitting diodethat is configured to emit red light.
 6. The display-pixel andphotosensor-element device of claim 1 wherein some of the plurality ofdisplay elements comprise a light-emitting diode that is configured toemit green light.
 7. The display-pixel and photosensor-element device ofclaim 1 wherein some of the plurality of display elements comprise alight-emitting diode that is configured to emit blue light.
 8. Thedisplay-pixel and photosensor-element device of claim 1 whereinlight-emitting diode comprises is an organic light emitting diode. 9.The display-pixel and photosensor-element device of claim 1 whereinlight-emitting diode is comprised by a stacked organic light emittingdiode (SOLED).
 10. The display-pixel and photosensor-element device ofclaim 1 wherein the light-emitting diode and the light-sensitive deviceare comprised by a stacked organic electroluminescent (SOE) structure.11. The display-pixel and photosensor-element device of claim 10 whereinthe light-emitting diode and the light-sensitive device are a singlestructure comprising full-color materials technology for RGB lightemission and RGB light detection, thereby integrating a photosensorelement and a display element into a single element.
 12. Thedisplay-pixel and photosensor-element device of claim 1 wherein thelight-emitting diode and the light-sensitive device are a singlestructure comprising full-color materials technology for light emissionand light detection, thereby integrating a photosensor element and adisplay element into a single element.
 13. The display-pixel andphotosensor-element device of claim 1 further comprising a plurality ofmicro-optic structures associated with at least the plurality oflight-sensitive devices.